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Abstract 

Can higher education institutions continue using planning 
and self-evaluation processes that are decades old without any 
reform? Several recent monographs urge institutions to embrace 
continuous improvement processes that engage systematic 
planning and department self-evaluation across whole campuses, 
but they do not show institutions how to do it. This article 
describes an improvement system shared by two very high- 
research universities that includes features such as interactive 
organizational learning environments, and it highlights the aspects 
of the system that help deal with some organizational problems 
such as administrative turnover, disconnected planning efforts, and 
limited sharing of information. 

Beyond TQM: Growing Expectations for 
Systematic Quality Enhancement 

While colleges and universities have been evaluating 
themselves for decades, the traditional processes — ad hoc 
assessments, program review, and accreditation self-studies — 
still suffer from structural negative factors such as administrative 
turnover, episodic self-evaluation, disjunctive planning efforts, 
department isolation, and the use of individual faculty vitae for 
demonstrating department-level impact. These factors frustrate 
the public's concern for accountability, retard the cycles of 
institutional improvement, and reinforce what Joseph Burke (2007) 
calls "the fragmented university." The next leap forward in higher 
education effectiveness will require campuswide structural and 
technological integration of planning and evaluation activities 
with growing numbers of faculty interacting in continuous 
improvement enterprises. 

Significant discussion about comprehensive continuous 
improvement systems for higher education institutions emerged 




© Copyright 2010, Association for Institutional Research 




Page 2 



AIR Professional File, Number 118, Improving the Way Higher Education Institutions Study Themselves 



/HR 

a * isaniiaii r a* 
InsHtutlanat Research 



in the early 1 990s. Lozier and Teeter (1 993) were 
among those who wrote about the application 
ofTotal Quality Management (TQM) principles 
to higher education operations. Among these 
principles are improving processes continuously 
and recognizing the university as a system with 
learning capability. Peter Senge (1990) also 
promoted the formation of organizational learning 
environments that help form collective realities, 
basing his organizational learning concepts on 
realistic self appraisal and continuous improvement 
(Lozier & Teeter, 1993). This approach makes 
process the focus and distinguishes TQM from the 
traditional higher education emphasis on outcomes 
assessment (Chaffee & Sherr, 1992). 

To apply TQM to institutions, Chaffee and Sherr 
stated that "the campus needs an infrastructure 
that will support increasing levels of integration, 
coordination, communication, and conflict 
management" (p. 90). The infrastructure must 
overcome the faculty members' perception that 
organizational service is an imposition on their time 
as independent entrepreneurs (Chaffee & Sherr, 

1 992). This often requires a change in culture, and 
an important step toward changing institutional 
culture is to establish an infrastructure that 
supports taking risks, learning from each other, and 
incorporating feedback (Freed & Klugman, 1 997). 

Among the barriers to implementing TQM on 
college campuses is the language ofTQM. There are 
pitfalls associated with emphasizing terms like quality 
management at institutions because they excite the 
faculty's fear of administrative over-centralization 
(Sherr, 1 991 , p. 4). Campuses that have avoided 
TQM terms and replaced them with action planning, 
feedback systems, or departmental impact and 
visibility indicators realize greater success. 

While TQM did not broadly capture the 
imagination of faculty or transform higher 
education institutions, it did call attention to 
the concept of establishing comprehensive 
and systematic infrastructures for accelerating 
improvement of programming. With recent 
technology innovations in database management 
and interactive Web resources, new energy is being 
invested into this idea. For example, TQM principles 



now appear in the recently revised standards of 
accrediting bodies. 

Several of the regional and special accrediting 
bodies in the United States revised standards and 
criteria and made them effective for institutions 
during the early 2000s. These revisions significantly 
shifted attention away from inputs and aspects 
of a compliance mode toward outputs and TQM 
characteristics. The Western Association of Schools 
and Colleges (2008) expects institutions to employ a 
deliberate set of quality assurance processes at each 
level of institutional functioning. These include 
assessing effectiveness, tracking results over time, 
using comparative data from external sources, and 
improving structures, processes, curricula, and 
pedagogy. The New England Association of Schools 
and Colleges (2005) asks institutions to work 
systematically and effectively plan, provide, oversee, 
evaluate, improve, and assure the academic quality 
and integrity of academic programs. Institutions 
located in the southeastern part of the nation are to 
engage in ongoing, integrated, and institutionwide 
research-based planning and evaluation processes. 

In addition to a transformation in standards, 
some regional accreditors developed alternative or 
new methods for institutional re-accreditation. The 
Higher Learning Commission's (2003) Academic 
Quality Improvement Program (AQIP) or the 
Quality Enhancement Plans (QEP) used by the 
Southern Association of Colleges and Schools 
provide institutions opportunities to focus on 
quality assurance and continuous improvement 
in priority areas, especially for student learning. In 
201 5, the Higher Learning Commission (201 0) plans 
to implement its new Pathways process, which 
includes a Quality Initiative requirement. 

Special accreditation organizations, such as 
the Association to Advance Collegiate Schools 
of Business International (AACSB), Accreditation 
Board for Engineering and Technology, Inc. (ABET 
), and the National Council for Accreditation of 
Teacher Education (NCATE), also used the early 
2000s to begin adjusting criteria to include tracking 
of student learning outcomes using evaluation 
processes that are systematic and continuous. The 
Teacher Education Accreditation Council (TEAC) 
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emerged in 1 997, in part, to offer teacher education 
programs an alternative way to continuously 
improve themselves based on planning and self- 
evaluation processes. 

Looking ahead to the future of accreditation, 
the Council for Higher Education Accreditation 
(CHEA) report stated that all accreditors will need to 
direct renewed attention to defining and assessing 
quality. The author of the publication, Peter Ewell, 
recommends that accreditors find ways to recognize 
exemplary performance by using multiple levels of 
accreditation that distinguish among different levels 
of quality assurance performance (Ewell, 2008). 

Several recent monographs also urge higher 
education institutions to embrace continuous 
improvement systems that engage systematic 
planning and unit self-evaluation across 
whole campuses. In 2006, Bok stated in Our 
Underachieving Colleges that 

Most successful organizations today, 
regardless of the work they do, are trying 
hard to become effective "learning 
organizations" that engage in an ongoing 
process of improvement by constantly 
evaluating their performance, identifying 
problems, trying various remedies, 
measuring their success, discarding those 
that do not work, and incorporating those 
that do. In theory, universities should 
be leaders in such efforts, since they 
have pioneered developing methods for 
evaluating other institutions in the society. In 
fact, however, they leave a lot to be desired 
when it comes to working systematically to 
improve their own performance, (p. 316) 

More recently, Burke (2007) suggests that unity 
can be put back into institutions by integrating 
processes such as strategic planning, priority 
budgeting, assessing institutional effectiveness, 
institutional accrediting, and internal performance 
reporting. He emphasizes the preservation of 
decentralization while developing centralized 
resources that provide institutional direction. 

With Honoring the Trust, Massy (2003) argued 
that more attention should be paid to developing 
quality processes, for these will deliver good 



outcomes. He reinforced this claim in Academic 
Quality Work (2007), stating that he had yet to find a 
department where systematic quality improvement 
efforts do not pay off. Earlier support for measuring 
process and developing organizational productivity 
came from Tierney (1 999). In The Future of Higher 
Education, Newman (2004) recommended that 
institutional leadership develop the capacity to 
draw the whole organization into the process of 
change, assessment, and constant and unremitting 
improvement. The academic unit best suited 
to collectivize faculty in these ventures is the 
department, says Zemsky (2005), changing quality 
efforts from individual, laissez-faire experiments to 
structured, monitored activities. 

These writers raise common themes that 
can help reform higher education: (a) use 
systematic continuous improvement systems, 

(b) integrate planning and funding efforts, (c) 
develop organizational learning resources for 
information sharing, (d) measure the effectiveness 
of improvement processes while monitoring 
participation, and (e) decentralize department 
activity but build centralized knowledge resources 
for improvement. This sounds good in theory, but 
does it work in practice? Where is it being carried 
out? What is its design? How well is it working? 

Quality Enhancement Systems 
and Their Design 

The following describes the components 
making up a universitywide continuous 
improvement system, reveals the benefits of 
inter-institutional collaboration in planning 
and self-evaluation, and discusses the impact 
of such systems. The opening section provides 
a description of the overall design of Colorado 
State University's (CSU) continuous improvement 
system with brief accounts of its 1 0 components. 
Discussion moves to a case-study examination 
of how the University of Nebraska-Lincoln (UNL) 
uses this system to form an organizational learning 
environment and affect faculty culture on its 
campus. An associate dean from CSU's engineering 
college adds a faculty perspective, addressing 
the challenges that a college faces when trying to 
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adopt a universitywide system. Near the end, the 
article describes the collaborative benefits that 
result when multiple institutions share a common 
assessment system. Evidence is provided to show 
the system's impact on faculty culture. 

Colorado State University: PRISM and 
A Big Picture Description of Continuous 
Improvement 

Colorado State University operates a continuous 
improvement system known as PRISM, or Plan for 
Researching Improvement and Supporting Mission. 
Nearly all of CSU's 1 69 undergraduate and graduate 
academic programs and 30 student affairs offices 
have developed assessment plans in the PRISM 
online database. The process generates hundreds 
of documented improvements in assessment, 
curriculum, faculty development, and student 



services each year. The 1 0 components of PRISM 
and their functions appear below. 

Component 1. Conceptual Model for 
Communication of Processes. A phrase, acronym, or 
design can help communicate the university plan, 
support the culture of improvement, and present 
the operational theme — decentralized planning 
and evaluation that leverages centrally developed 
resources to systematically manage change. The 
conceptual model presents a visual representation 
of the process, that is, a prism separating out the 
effort of faculty to evaluate student learning, 
research, and outreach into multiple uses of this 
information while forming an organizational 
learning environment that encourages sharing 
among faculty, students, and accrediting bodies 
(see Figure 1). 
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Figure 7. The PRISM model. 
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Component 2. A Building Platform for 
Annual Online Assessment Planning. Faculty 
use an interactive Web site to generate program 
planning templates, report self-evaluation data, 
access information sharing reports, and engage 
in interactive peer reviews. To view the main site 
and look at a demonstration of assessment plan 
building, see http://assessment.colostate.edu and 
use demo for the departmental username and 
password. 

Component 3. Active Timelines to Monitor 
Planning and Evaluation Process Steps. A planning 
timeline is displayed so that individuals can monitor 
institutionwide participation levels. This timeline 
also organizes activities into discrete steps (see site 
and use demo password referenced in Component 2 
above). 

Component 4. Selecting and Configuring 
Evidence Using Classification Processes. A 

classification process within the database helps 
generate multiple evidence presentations of 
programs' self-evaluation findings. This is done to 
maximize faculty effort for satisfying varying quality 
monitoring requirements — state accountability, 
regional and special accreditation, institutional 
program review, and response to market forces. 
Using the access to PRISM defined in Component 
2 above, select "View Department and College 
Assessment Output Reports" and select either 2004- 
2005 or 2005-2006 timelines to review presentation 
formats. 

Component 5. Aligning Assessment Plans 
with Program Review Self-Studies to Give Added 
Visibility to Student Learning Outcomes Research. 

A structured planning integration within a single 
database is more likely to sustain faculty interest 
in learning assessment over time and encourages 
planning coherency, which helps accelerate 
program improvement. Multiple reviewing 
audiences are repeatedly exposed to the best 
practices in learning assessment and departmental 
self-evaluation processes that are embedded into 
the online planning templates. 

Component 6. Multiple Peer-Review Levels 
Enhance Compliance with Quality Standards. An 
institutional peer-review process encourages the 



quality enhancement of departments' planning 
and self-evaluation because of the uniform and 
evolving standards of best practice as articulated in 
an institutional rubric of standards. See "Academic 
Peer Review Rubric" using the access instructions 
in Component 2 above. There is also a one-page 
excerpt of the rubric in Table 1 . These Best Practices 
were developed through repeated peer-review 
readings of assessment plans, where a committee of 
faculty and an assessment director recognized and 
recorded program practices that went beyond the 
university standards. 

Component 7. Documentation of 
Interactive Conversations Among Faculty 
About Improvement. Interactive online dialogue 
among faculty that is embedded in assessment 
plans and program review self-studies enables 
external constituents to learn the effectiveness 
of an institution's self-evaluation process, the 
rate of continuous improvement, and depth of 
organizational learning. 

Component 8. Sharing Best Practices and 
the Evaluation Forms Faculty Use to Research 
Learning. Campuswide sharing of instruments 
for measuring unit performance can contribute to 
the consideration of various alternatives and can 
improve unit planning and self-evaluation methods. 
See "View Evaluation lnstruments"and"View Best 
Practices" using the PRISM access instructions in 
Component 2 above. 

Component 9. External Engagement Web Site 
for Multiple Audiences. A public Web site, "Planning 
for Improvement and Change," provides added 
transparency of the process. It presents institutional 
performance research and program improvements 
to multiple audiences — students, faculty, parents, 
employers, accrediting teams, and state policy- 
makers using tailored pathways of discovery. For 
example, students can use the "PRISM Archive 
for Public Access" to view program-level learning 
assessment plans (see http://improvement. 
colostate.edu). 

Component 10. Reports Generated 
on Planning and Self-Evaluation Process 
Effectiveness. Annual reports generated from 
PRISM inform faculty and administrators of the 
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Table 1 

CSU Peer-Review Committee Rubric for Evaluating Program Assessment Plans (one-page 
excerpt) 



Program Plan 


PRISM Performance Levels 


Components 


Best Practices 


Well Developed 


Under-Developed 


Assessment 


25. Program uses course-based research 


Method satisfies at least traits A, 


A. Method does 


Method(s) 


related to formative assessments such as 


B, C & D 


not satisfy trait A or 




pre-instruction questionnaires, minute 




satisfies two or fewer 


PART III 


papers, meta-cognition exercises, concept 


A) At least two outcomes have 


of the B, C, D traits in 




mapping and others to affect instruction 


used "DIRECT" assessment 


"Well Developed." 




and learning during the course. 


(a student demonstration of 
learning evaluated by faculty 






26. Graduate program uses direct 


or persons in addition to the 






assessment or student demonstrations 


course instructor). 






to research the effectiveness of faculty 


-—It is OK for one outcome to 






mentoring of graduate students in 


use indirect assessment as the 






preparing them for the field's expected 


primary assessment. Indirect 






profession characteristics, such as personal 


includes survey, focus group, 






behavior, knowledge of publishing 
process, professional development 


interview or others. 






knowledge, conference proposals, 


B. Assessments measure each 






association membership, and others. 


learning characteristic listed in 
the outcome (e.g., for writing 






27. Program uses a pre-instruction 


it might include measures for 






questionnaire to attain incoming student 


1) grammar, 2) organization, 






knowledge profiles, e.g., determine 


3) thesis development, 4) 






confidence levels in specified knowledge 


documentation, 5) critical 






or skill areas and/or be informed 


thinking synthesis) to achieve 






of knowledge misconceptions or 


multiple measures that 






remediation needs. 


can reveal strengths and 
weaknesses. 






28. Program formally shares its learning 
research data with other departments 


C. Method describes how 






that have students or majors engaging its 


faculty will operate the student 






curriculum, e.g., CMATE survey results sent 


demonstration or conduct a 






to Teacher Licensure. 


survey and how faculty will 
gather the data (sample size 






29. Program uses its assessment process 


of demonstrations — 20% of a 






as an instructional method to promote 


1 00 student essays ) and how 






active learning of outcomes, and instill life- 


faculty will conduct the scoring 






long professional behavior of conducting 


(rating system such as Likert, 






self-evaluation research_e.g., students use 


size of faculty group and use 






program learning rubric to evaluate each 


of scoring instruments such as 






other's work/project or their own. 


rubrics). 






30. Program uses a learning research 


D. Describes how assessment 






method that aids student transition to 


results will affect decision 






the workplace, e.g., a portfolio of works, 


making for the unit (how 






audition demonstration, or other world of 


results are used to improve 






work-based demonstrations that students 
can apply in the hiring process. 


programming). 
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systemwide aggregated outcomes or improvements 
at the department, college, and university levels. 
These include assessment profiles (what is a 
department or college measuring and what are 
its data-gathering methods) and the quality of 
programs' planning and self-evaluation processes. 
Examples of these reports on assessment can be 
viewed using the web access instructions contained 
in Component 4 above. 

Four Central Themes of the PRISM System 
and Contribution of the Ten Components 
Theme 1 — Sustainability and Visibility. 

Component 1 (conceptual model) is designed to 
develop the enduring image or concept along 
with a common phraseology that creeps into 
institutional policy language and memory over 
time. For example, at CSU, the word PRISM appears 
in the new institutional strategic plan as well as in 
several of the college strategic plans. Component 
2 (planning platform) is intended to establish 
routine behaviors of annual reporting and data 
gathering until they become departmental habits. 
The quality of assessment planning is regularly 
evaluated using Component 6 (peer review). It 
exposes any backsliding on institutional standards, 
such as failure to use direct assessment. The review 
process works to re-establish good processes. If 
departments fall behind the annual timeline, 
Component 3 (monitoring), which tracks progress, 
will help identify lagging units with significant 
visibility, giving administrators the information they 
need to send units repeated reminders to catch 
up. Faculty members remain interested in their 
annual assessment activity because Component 
5 (connecting assessment and program review) 
automatically embeds their program assessment 
plans within the institution's online program review 
self-study process. Assessment of learning receives 
greater exposure across campus and becomes a 
higher stakes activity as multiple faculty groups 
or review entities critique each department's 
learning outcomes research. Because the feedback 
comments of reviewers are documented within 
the embedded assessment plans and within the 
self-study itself, a record of improvement dialogue 



emerges. A combination of high visibility and 
peer pressure draws more attention of responsible 
individuals to learning assessment. 

Theme 2 — Planning Context and Integration. 
Assessment planning at CSU has become a 
departmental, annual operational planning exercise 
for learning, faculty research, and faculty service/ 
outreach. Components (connecting assessment 
to program review) permits external reviewers to 
learn about a department's planning and self- 
evaluation effectiveness overtime, which can be 
more revealing than analyzing only institutional 
research data — the number of majors or volume 
of student credit hours produced. Linkage to 
institutional strategic planning is accomplished 
through the self-study action plan development. 

The database embeds the University strategic plan 
into the action plan goals and requires departments 
to use a check-box exercise to relate the strategic 
goals and metrics to each action goal before text 
can be entered. Both academic departments and 
student affairs at CSU use both the assessment plan 
and program review parts of the PRISM database, 
enabling the institution to track planning activity 
devoted to each institutional strategic planning goal 
and objective. Persons across campus can use the 
database to form reports on planning integration 
by department or college level, making strategic 
planning implementation more visible. In addition, 
the institutional peer-review process for assessment 
(Component 6 — one committee for academic 
plans and one committee for student affairs plans) 
annually monitors departmental plan development 
so that the unit planning supports university-, 
college-, and division-level strategic planning goals. 

Theme 3 — Organizational Learning 
Environment. An institution can delegate the 
peer-review process (Component 6) to one or more 
committees. These groups work to actively guide 
or direct the planning design so that the planning 
generates evidence (Component 4 configuration) 
needed by (a) regional or special accreditation 
bodies, (b) the institution's strategic planning 
metrics, or (c) state accountability expectations. 

The accrediting criteria are contained within 
the Academic Peer Review Rubric referenced in 
Component 6. 
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Using their embedded feedback comments 
in unit assessment plans, peer review members 
regularly encourage programs to use the PRISM 
planning for discovering performance strengths and 
weakness, not to just confirm or monitor activity 
levels. The review experience is designated as formal 
leadership training for faculty, because committee 
members are provided the opportunity to become 
experts in the best practices of departmental 
planning, self-evaluation, organizational 
management, and systematic improvement. The 
peer-review work of identifying best practices and 
giving constructive feedback to programs on how 
to improve planning and evaluation processes 
helps generate an online organizational learning 
environment where the institution's study of itself 
is analyzed and discussed. Analysis of several 
cycles of program review self-studies reveals that 
these studies improve over time — becoming more 
substantive — because the online template includes 
recent CSU best-practice descriptions throughout. 
These descriptions present knowledge of effective 
department organizational strategies for improving 
learning, research, and service to different faculty 
each year as multiple program review teams work 
through the self-studies. Component 8 (resources) 
adds to campuswide learning by providing access 
to hundreds of learning evaluation forms and 
departmental research/service quality definition 
documents. Therefore, academic departments can 
learn from each other about how to systematically 
develop research and leadership impact in their 
respective disciplines. The strong performers can 
help others and accelerate the rate of institutional 
improvement. 

Finally, Component 9 (transparency) works to 
attract students, parents, accrediting bodies, and 
others into the learning environment. Information 
from the PRISM process is shared with multiple 
audiences, some for marketing purposes. Each 
talking point about the institution must be 
supported by assessment evidence. For example. 
Education Benchmarking Inc. data supports the CSU 
claim of scoring in the top 25% of its institutional 
peers for safety and security in its residence halls. 
Students can access the learning outcomes and 



assessment methods for all academic programs, 
undergraduate and graduate. Accrediting bodies 
are guided to program improvement data, which 
supports Component 4 (selecting and configuring 
evidence). 

Theme 4 — Reporting System Output. An 

institution can claim it has a legitimate continuous 
improvement system only if it can show the 
impact and output of that system. Initially, 
central administrative staff members developed 
a taxonomy of planning and self-evaluation 
activity and embedded it into plans contained 
in the assessment database. Individual planning 
sections are classified to configure evidence for 
external quality monitoring (Component 4) and to 
generate reports (Component 10). These reports 
show the assessment profiles of the institution, its 
colleges, and departments (e.g., types and number 
of learning outcomes, the types and number of 
learning research methods, and types and number 
of improvements). 

Component 10 also includes a planning 
effectiveness index that describes the quality 
of departments' planning and self-evaluation, 
using 16 individual indicators of planning and 
evaluation effectiveness (see Table 2). Classifying 
capacity permits staff to document performance 
on each indicator for all programs in the database. 
For example. Indicator 6 (diagnostic capacity) is 
checked for a program's assessment plan when staff 
recognize that its data findings reveal the strengths 
and weaknesses in student learning performance. 
For example, some programs use a detailed, 
primary-trait rubric that their faculty developed to 
evaluate a research paper and then diagnostically 
report that 80% of students are proficient in writing 
organization, 88% are proficient at originality, 91% 
are proficient at grammar/mechanics, and that 58% 
are proficient at synthesizing references effectively. 
Poor-performing programs use weak definitions of 
learning expectations and provide just one holistic 
score, such as 75% of students are proficient in 
writing, which is not diagnostic because there is no 
way to guide writing improvement. 

Definitions of some of the other 1 6 effectiveness 
indicators follow. Indicator 1 . Range of Outcomes 
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Table 2 

Planning and Evaluation Effectiveness Index: Annual Report for 2003-2004 





INDICATOR 


ART 


ENG 

LISH 


FOR 

LANG 


MUSIC 


PHIL 


SPEECH 


AVG. 

HUM. 


Student Learning 


Indicator 1 Range of Outcomes 


5.0 


3.0 


3.0 


4.0 


3.0 


4.0 


3.7 


Indicator 2 Intensity of Exploration x .50* 


4.9 


4.8 


6.7 


4.6 


5.9 


5.8 


5.4 


Indicator 3 Measuring Methods 


6.0 


9.0 


4.0 


6.0 


5.0 


4.0 


5.7 


Indicator 4 Measuring Frequency 


4.0 


11.0 


4.0 


3.3 


5.0 


3.0 


5.1 


Indicator 5 Measuring Points in Time 


3.0 


2.0 


1.0 


3.0 


1.0 


2.0 


2.0 


Indicator 6 Diagnostic Capacity 


1.6 


2.3 


2.3 


0.1 


1.5 


1.0 


1.5 


Indicator 7 Improvemaent Range 


2.0 


1.0 


4.0 


2.0 


0.0 


4.0 


2.2 


Indicator 8 Improvement Frequency 


1.2 


2.0 


2.5 


1.7 


0.0 


3.0 


1.7 


4-> 

si 

& U 


Indicator 9 Feedback-best practices 


4.0 


3.0 


2.8 


0.3 


1.0 


3.5 


2.4 


Indicator 1 0 Feedback-comments x .50 * 


-5.6 


-6.3 


-8.6 


-9.4 


-5.8 


-10.0 


-7.6 


to U) 
>\ 03 
zj) 


Indicator 1 1 Low Participation Level 


0.0 


0.0 


0.0 


-1.4 


-1.5 


0.0 


-0.5 


c 

LU 


Indicator 12 Low Participation Level 


0.0 


0.0 


0.0 


-1.6 


-1.5 


0.0 


-0.5 


Research & 
Service 


Indicator 13 Range of Exploration Research 


3.0 


2.0 


2.0 


4.0 


2.0 


4.0 


2.8 


Indicator 14 Use of Impact Indicators Research 


3.0 


0.0 


3.0 


1.0 


0.0 


4.0 


1.8 


Indicator 1 5 Range of Exploration Service 


4.0 


2.0 


2.0 


3.0 


1.0 


0.0 


2.0 


Indicator 16 Use of Impact Indicators Service 


1.0 


0.0 


1.0 


0.0 


0.0 


0.0 


0.3 


Indicator Groups 


A) Performance Research Range/Intensity: 


16.9 


11.8 


0.0 


15.6 


11.9 


13.8 


13.9 


B) Measuring Frequency: 


13.0 


22.0 


13.7 


12.3 


11.0 


9.0 


12.7 


C) Changes/Improvements: 


3.2 


3.0 


9.0 


3.7 


0.0 


7.0 


3.9 


D) Diagnostic Capacity: 


5.6 


2.3 


6.5 


1.1 


1.5 


5.0 


3.6 


E) Peer Review Feedback: 


-1.6 


-3.3 


6.3 


-9.1 


-4.8 


-6.5 


-5.2 


F) Reporting Participation: 


0.0 


0.0 


-5.8 


-3.0 


-3.0 


0.0 


-1.0 


Department Level 
EFFECTIVENESS / INDEX SCORE » 


37.1 


35.8 


29.7 


20.6 


16.7 


28.3 


28.0 



Source: Bender, Kim. (2003) Plan for Researching Improvement and Supporting Mission. Colorado State University 

• Indicators 2 and 1 0 include weighting features that reduce their impact on the index. If left with its full weight, indicator 2 
would reward programs that merely write a good research plan or intend to study in depth, but do not gather any data and 
do not report findings. Additionally, while peer-review feedback is defined as a negative indicator as they only appear if 
something is wrong with a plan, the system does not want to inhibit the peer-reviewers from making comments in the plans. 

• Explanatory Note: Table shows Philosophy as performing at a lower level (16.7) with weakness in the areas of improvements 
(7,8) and low participation (11,12). 



relates to the breadth of learning research — 
whether a program only researches writing skills 
or if it also measures problem solving, project 
planning skills, application of knowledge, or others. 
Indicator 2. Intensity of Exploration relates to the 
depth of research — whether a program researches 
writing holistically or if it also measures the 
subcomponents of writing, such as organization, 



thesis development, transitions/flow, mechanics, 
synthesis, documentation, and others. Indicator 
3. Measuring Methods relates to the variety of 
instruments used to gather data — whether a 
program uses only a graduating exit survey or 
whether it also uses a culminating project with 
a rubric and uses an internship form for skills 
application, and uses a pretest to assess incoming 
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skills and others. Indicator 4. Measuring Frequency 
relates to how many times the program uses its 
assessment instruments in an academic year (e.g., 
once per year or once per semester or more). 
Indicator 5. Measuring Points in Time relates to how 
many times during a student's career the program 
makes measurements of learning (e.g., beginning, 
middle, end, and postgraduate). Indicator 7. 
Improvement Range relates to the number of 
program categories improvements cover (e.g., 
assessment, curriculum, professional development, 
technology, or others), while Indicator 8. 
Improvement Frequency refers to the number of 
improvements implemented per year. A program 
earns negative numbers if it receives repeated and 
excessive peer-review Feedback (Indicator 9) and 
if it fails to Participate (Indicator 1 1 ) or report data 
findings. For faculty research and service, programs 
earn positive numbers if they demonstrate planning 
that uses Impact Indicators (Indicator 14), such 
as a citations index for publications or volume of 
leadership positions on editorial boards, grant 
review boards, or professional associations. A 
complete analytic indexing table is available from 
the first author. 

Added together, these 1 6 indicators comprise a 
single process-effectiveness index. The index serves 
as an overall diagnostic indicator in that its measure 
of process and activity identifies potentially low- 
performing departments and reveals what process 
areas seem to most require improvement (see Table 
2). These department-level planning and evaluation 
effectiveness reports are automatically embedded 
in departments' online program review self-studies 
as a resource for evaluation and to provide visibility 
of performance. 

In other words, as a good continuous 
improvement system should, PRISM regularly 
informs the institution what processes to improve 
to upgrade how well it studies itself. The PRISM 
system focuses on improving processes more than 
on checking whether or not program outcomes 
are attained. This discussion has shown how the 
1 0 Components interact to produce synergies of 
improvement at CSU, how they support the four 
themes, and how PRISM is consistent with the major 



principles of TQM, but can PRISM demonstrate 
portability and function elsewhere? The following 
demonstrates the potential of PRISM at other 
institutions. 

System Implementation: Case Study 
from the University of Nebraska 
at Lincoln 

Introduction 

A key reason for the University of Nebraska at 
Lincoln (UNL) to implement CSU's PRISM system 
was to foster faculty leadership and create faculty 
ownership for a process of continuous program 
improvement that values evidence about student 
learning outcomes. The best opportunity that 
assessment has for improving student learning 
happens when those who are responsible for 
making decisions about the future direction of 
curriculum and instruction value evidence of 
learning as a means of informing those decisions 
(Banta, 2002; Maki, 2004; Schilling & Schilling, 1 999; 
Wergin, 2000). We used the key components of the 
CSU PRISM system and adapted it to our situation. 
The acronym we use is PEARL (Program Excellence 
through Assessment, Research, and Learning). This 
case study describes UNL's implementation of 
PEARL and how this implementation is encouraging 
a cultural shift among faculty, not from the top 
down but up from the broad base of faculty 
members. 

Hallmarks of Faculty Engagement 

At one time, like many institutions, UNL 
complied with basic expectations of outcomes- 
based assessment but had not fully committed 
to systematic continuous improvement based on 
evidence. In stating the problem of integrating 
assessment into the culture of institutions 
Wergin (2000) identifies a key issue affecting 
this commitment. He states that "faculty accept 
the necessity of program review and outcomes 
assessment, but don't generally see how these 
processes will affect their own professional practice, 
at least not in a positive way" (p. 7). This reflects 
the experience of many institutions. Academic 
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programs gather information about student 
learning and report that information with the 
purpose of program improvement, but it tends to 
be a separate add-on exercise rather than a routine 
and natural part of program functioning. The ideal 
environment would have the following hallmarks 
(Banta, 2002; Maki, 2004; Schilling & Schilling, 1 999; 
Wergin, 2000): 

• Programs would continually engage in this 
process of inquiry and build upon what they do 
and learn overtime. 

• Responsibility would not fall to a single 
individual but instead there would be collective 
responsibility to the process by a critical mass. 

• Faculty and their programs would engage 
in outcomes-based assessment because it 
is valued internally instead of being seen as 
exercise expected by external audiences. 

• Discourse about what students are learning and 
how to best represent learning would occur 
naturally without prompting among faculty 
across the program. 

• Mechanisms for documenting the process 
would make evidence readily available and 
easily accessible to programs and the institution 
for multiple reporting purposes. 

CSU's PRISM system provided the potential 
for progress in each of these areas. The system's 
features not only provided essential components 
for assessment but also provided the opportunity 
to accelerate our institutional understanding about 
effective structures and supports for outcomes 
assessment by entering into a partnership with 
a similar institution to share and collaborate on 
approaches and lessons learned. 

The 1 0 components comprising PRISM offer 
institutions like UNL the flexibility to implement 
only those that match its priorities. Initially, UNL 
used the following components: concept model 
(Component 1), planning platform (Component 
2), timeline monitoring (Component 3), peer 
review (Component 6), documenting faculty 
dialogue (Component 7), and sharing best practices 
(Component 8). Shortly after implementation, UNL 
began using the classification and configuration 
features (Component 4) along with reporting 



features (Component 10). The remaining two 
parts, connecting assessment to program review 
(Component 5) and using a public transparency 
site (Component 9), are being explored and are 
under discussion. The use of the system permits 
institutions to expand components as the faculty 
culture on campus evolves towards a more 
comprehensive continuous improvement process. 

Organizational Structure: Creating 
Leadership and Effective Environment 

Faculty members are a central focus of PEARL'S 
organizational structure and consist of three key 
groups: Peer Reviewers, Program Leaders, and a 
Steering Committee. Peer Reviewers are faculty 
members who are seen as leaders in their college, 
showing an interest in the development of both 
their programs and their faculty colleagues. The 
associate deans of the colleges, who comprise the 
membership of the assessment steering committee, 
identify and encourage faculty to serve on the 
Peer Review Committee. This group consists of 
1 5 faculty members who work in small, cross- 
disciplinary teams. Its key activity is to review each 
program's assessment and results and provide 
feedback. To prepare them for this role, they are 
involved in a series of workshops about assessment 
and about the development and application of the 
PEARL rubric for evaluating programs'assessment 
processes. Initially, a collaboration among the 
university's assessment Steering Committee, 
the Peer Review Committee, and the Director of 
Assessment developed the PEARL quality standards 
rubric that describes the institution's quality 
expectations for program assessment planning 
and self-evaluation of student learning. Each 
year the peer-review members update the rubric 
traits. A single qualitative rubric helps the Peer 
Reviewers establish continuity and integrity to the 
evaluation process, while it assists the training of 
new Peer Reviewers (http://www.unl.edu/ous/pearl/ 
Resources.shtml). 

The Peer Review committee's work is essential to 
develop a stronger faculty culture for assessment. 
How did this work? By practicing multiple reviews 
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over a three-year committee term, these individuals 
formed a relatively sophisticated understanding of 
purposes and approaches to assessment that could 
be translated for their colleagues. Using the context 
and language of their disciplines, they help their 
colleagues realize how systematic improvement 
activity could help them address the program issues 
they cared about and ones that affected their daily 
work. 

Program Leaders are faculty members 
appointed by their department chairs to plan 
and implement evaluation methods and to 
document their program's assessment activity in 
the system's database. Often, these individuals are 
undergraduate or graduate curriculum committee 
chairs. These individuals are involved in a series of 
workshops training them on the use of the online 
system and administration of assessment planning. 

A third group, the Steering Committee, 
involves associate deans from each college, the 
Director of Institutional Assessment, and an 
Assessment Associate for PEARL. This group plans 
the implementation of the system and continues 
to guide its progress as a whole. Members 
work toward a shared institutional vision. They 
sustain the decentralized nature of specific 
assessment content and implementation, while 
using the common database structure to develop 
universitywide information that is organized to 
inform broad planning efforts. 

System Facilitation of the Culture 
Transformation Goal 

PRISM's 10 components encouraged movement 
toward our goal of developing a faculty culture 
around assessment. The contributions of the PRISM 
components implemented by UNL are described 
below. 

7. Implementation of PRISM Components 1, 

2, and 3: Creating a shared understanding and 
common documentation structure. Each year, 
programs select three program student learning 
outcomes to assess. Documentation consists of six 
planning aspects reported in a two-cycle process. 

In the first cycle, programs define and report on four 



aspects of their plan: learning outcome, student 
opportunities to learn the outcome, a question 
about the learning outcome that is of interest to 
the program, and the assessment method used to 
research the learning outcome. Programs receive 
feedback on these four aspects from Peer Reviewers 
before implementing their plans. Peer Reviewers 
use the PEARL quality standards rubric descriptions 
for each component as a guide for their feedback. 

In the second cycle, after completing the 
assessment, programs summarize and report two 
more aspects from their assessment: (a) the results 
of their research of the learning outcome, and 
(b) their planned use of those results, including 
program improvements. Again, programs receive 
feedback on their data findings from the Peer 
Reviewers, who use the PEARL rubric as a guide. 

This structure has several benefits. The common 
documentation structure facilitates the compiling 
of information over time and across units. It also 
clarifies expectations of important steps in the 
assessment process and provides a familiar format 
for viewing what others are doing. 

Having this information online further 
strengthens institutional planning effectiveness. 

It allows programs accessible documentation 
that aids continuity of faculty and staff effort, 
despite personnel turnovers. It also accumulates, 
in the short term, evidence that a program needs 
to demonstrate outcomes in the long term. By 
focusing on only three outcomes each year, the 
process of assessing all relevant program learning 
outcomes is spread over time and is therefore made 
more manageable. Over a period of five to 1 0 
years, programs will accumulate a wealth of broad, 
in-depth documentation about their program's 
planning and self-evaluation effectiveness. This will 
be valuable for strengthening academic program 
review, accreditation, or strategic planning. 

The six aspects, particularly the "question of 
interest," suggest a research orientation that helps 
faculty see the parallels between the processes 
of assessment and their own discipline-based 
research. It is important that faculty be provided a 
"foundational basis for effective practice" because 
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the use of information is a "learned skill" (Wergin, 
2000, p. 23). The "question of interest" section also 
makes a statement that assessment should be 
directed by pressing questions that faculty have 
about their programs and their students learning. 
This helps achieve the desirable goal where 
assessment efforts of faculty are driven by their 
questions, not focusing as much on the assessment 
design or process details as much as determining 
how the assessment findings are to be used (Banta, 
2002; Wergin, 2000). 

2. Implementation of PRISM Components 6 
and 7: Facilitating faculty dialogue about effective 
assessment practice. As previously mentioned. Peer 
Reviewers provide feedback to programs on their 
assessment plans and on their evaluation results. 
The system facilitates this feedback by providing 
interactive online dialogue boxes for Peer Reviewers 
to enter their comments and for Program Leaders 
to respond to those comments. Peer Reviewers' 
feedback comments are guided by the PEARL 
rubric that identifies effective characteristics of 
assessment for each of the six planning aspects. 
These standards provide other benefits as well. 

The Peer Review Committee uses the PEARL rubric 
to annually identify best practices and to hone 
new incoming best practices as they emerge from 
faculty experiences. Programs use it to learn what 
an effective assessment process should look like. 
Regular use of these standards provides an efficient 
way to administer continuous quality enhancement 
of the program assessment process. 

The PEARL process also accelerates the 
understanding and use of better approaches. 

For example, in our first year the use of direct 
assessment increased from 11% of the assessment 
methods used to 75% of the assessment methods 
used in the second year. The venue for faculty 
dialogue also provides an organizational learning 
environment that Wergin (2000) recommends 
because it provides a climate supportive of quality 
improvement and treats the department as a 
collective body. The system prompts faculty from 
different disciplines to discuss definitions and 



representations of student learning. For the 
institution, capturing this dialogue about quality of 
assessment and definition of learning can be quite 
useful for demonstrating faculty engagement in this 
process for external audiences. 

3. Implementation of PRISM Component 
8: Providing access to methods and measures. 

As Program Leaders enter their planning and 
evaluation information online, they are able to 
upload supporting documents for their assessment 
process including research instruments, surveys, 
rubrics used in their assessment method, result 
tables, or data graphs. These products are then 
classified by type and made accessible to all 
programs on the system. This access is beneficial 
because it provides a cross-fertilization of ideas 
among various disciplines. In the first year, over 

1 00 assessment documents were uploaded to the 
PEARL site. This volume of tools for researching 
student learning is then made available to all 
programs within and across the two collaborating 
universities. This access demonstrates that there 
is no one best measure or approach and gives 
programs a better opportunity to find methods that 
fit the scholarship of their discipline. 

4. Implementation of PRISM Component 8: 
Raising expectations for high quality processes 
with examples of best practices. This system 
highlights "best practices" emerging in each of 
the six aspects of programs' plans (e.g., outcomes 
or assessment methods). Once a best practice is 
highlighted for an aspect, a description of why it 
was identified as a best practice is provided, and all 
faculty from all the different programs in the system 
can view the actual best practice and why it was 
selected. This expands the resources available to 
programs by making available alternative practices 
that demonstrate the application of planning and 
effective self-evaluation methods. Over time, this 
contributes to raising the level of quality across all 
programs. In interviews conducted with Program 
Leaders, some respondents indicated they found 
the PEARL peer-review rubric a helpful resource in 
their program's assessment planning. Others 
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preferred directly viewing other programs' practices. 
Results of these interviews suggested that it was 
important to have both types of resources available 
for Program Leaders to use when planning their 
program's assessment process. 

5. Implementation of PRISM Components 
4 and 10: Capturing and compiling activities 
and results across units. A classification feature 
allows the identification of the types of assessment 
activities and categorizing progress within and 
across programs. A central unit, such as an 
institutional assessment or research office, can 
use this to develop profiles of an institution's 
assessment activity as well as produce departmental 
reports showing planning participation and 
assessment effectiveness. For example, one 
can classify the types of learning outcomes that 
programs are selecting, the types of measuring 
methods used (direct or indirect), how results are 
being used, and types of program improvements 
implemented as a result of assessment. Once 
classifications are determined, information can 
be compiled about each of those categories at an 
institutional, college, or program level. A program 
might compile such information over several years 
for an academic program review or accreditation 
visit. An institution might compile that information 
over all units to track and demonstrate progress 
and impact of assessment across the institution. 

This feature allows documentation to be entered 
into the system over time and to be compiled at 
the appropriate level, when needed, for a variety of 
reporting requirements. In addition, since activities 
and program self-evaluations are captured each 
year, the system has the potential to demonstrate 
continuous and evolving judgments of a program's 
effectiveness over time. The classification scheme 
can be tailored to effectively communicate 
progress that addresses specific reporting criteria 
or expectations, and it can be used to compile 
this information quickly. This type of long- 
term evidence is not only important to external 
audiences for accountability, but it is especially 
valuable for convincing internal audiences such as 
our own faculty and administrators that planning 
and assessment are worth their time and effort. 



The Perspective of an 
Associate Dean: Challenges 
of Adopting a Universitywide 
Improvement System 

The previous two discussions have been 
about implementing a planning and assessment 
process from the perspective of the institution. The 
implementation of a system, such as PRISM, at the 
college level presents several specific challenges. 
The following is a discussion of the issues of 
implementing a planning and assessment system 
from the viewpoint of the College of Engineering 
(Engineering) at CSU. 

The connection between university priorities 
with college and department priorities is not always 
as clearly seen from the college level as it is from 
the university level. As you get closer to the faculty 
level for implementation, questions arise as to 
why an additional activity is more important than 
the other many demands on faculty time? These 
provincial perspectives must be overcome before 
the implementation can proceed. Adding to the 
complexity, Engineering at CSU considers specialty 
accreditation to be the most important motivator 
for assessment programs; therefore, broader 
assessment programs are perceived to be a greater 
burden than those already in place for accreditation 
and are quickly rejected or resisted. 

Our goal in this section is to illustrate how 
these issues have affected the adoption of the 
PRISM system by Engineering at CSU. As the PRISM 
system was first being developed, each college 
provided a point person to coordinate the effort — 
for Engineering, this was the Associate Dean for 
Academic and Student Affairs. This coordinator has 
the job of bridging the gap between the perceived 
university-level priorities and those of the college 
and departments. Success in this endeavor requires 
that tangible reasons for aligning the priorities can 
be developed. 

As we started this process of moving 
Engineering towards implementing PRISM, the 
following perceptions arose, some of which still 
exist: 
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• We already do this — leave us alone; 

• We can do it better; 

• It is unnecessary; and 

• It is another fad soon to be forgotten. 
Undergraduate programs in engineering colleges 
are accredited through the Accreditation Board 
for Engineering and Technology (ABET). Starting 
in 2000, ABET accreditation required programs to 
measure the attainment of student outcomes — 
similar to the requirements supported by PRISM. 
Therefore, CSU engineering faculty believed there 
was no need to shift efforts to using the university 
system, when they already were accomplishing 
this same objective in their own unique manner. 

And in fact, we claimed that we had been doing it 
longer and, therefore, had a better approach. This 
rendered PRISM unnecessary, because its only effect 
would be to require duplication of existing efforts. 
And to top off the arguments against PRISM, one 

of the oldest roadblocks used to resist university 
initiatives was put up: "This is just a fad that will 
soon pass as soon as a new administration takes 
over." All of these arguments provided reasons not 
to change current behavior. 

Even though each of these arguments can be 
refuted, this still does not automatically result in 
acceptance. First, the arguments that engineers 
already do assessment and are probably better at 
it than the university can be addressed. Although 
engineering programs are required to assess 
student outcomes, most have relied almost 
exclusively on survey instruments that provide 
indirect measures of outcome attainment. When 
ABET first propagated new accreditation criteria 
concerning the assessment of student outcome 
attainment, it recommended that programs use 
both direct and indirect measures of assessment. 
Unfortunately, using direct measures was not 
the norm in engineering, so adoption of this 
recommendation has been slow in coming. 

The result is that ABET has been very forgiving 
in pushing this agenda. Programs outside of 
engineering, which do not have a tradition of 
assessment, are free from this obstacle of replacing 
an existing approach — instead they must be 
convinced to try something new — which may be 
easier! 



The necessity of meeting the requirements 
propagated by PRISM seem remote to Engineering, 
where specialty accreditation tends to take 
precedence over other external pressures. What 
this perspective fails to recognize, however, is 
the importance of regional accreditation, which 
has become very aggressive in pushing for direct 
measures of student outcomes. This regional level 
of accreditation is less forgiving than engineering 
accreditation (although engineering accreditation 
continues becoming more rigorous each year) when 
forcing this change in assessment methodology. 

The national context of program assessment has 
changed dramatically in terms of the calls for 
greater accountability of student learning from 
constituents outside the academy. When one takes 
this greater context into account, PRISM can be 
viewed as a comprehensive approach for addressing 
these growing calls for program accountability. 

The key is to educate the college and departments 
about this context, which will continue to grow in 
importance for higher education. Therefore, the 
perception that this is just a local administrative 
fad that will disappear can easily be refuted if these 
units value their external constituents. 

With all this working against the adoption of 
PRISM by Engineering, some success has been 
realized. When the university first pushed for 
the use of PRISM, Engineering developed an 
initial set of plans (Component 2 platform). These 
plans reflected the current state of assessment 
in the college more than any attempt to develop 
significantly new approaches. The PRISM process 
previously described resulted in the identification of 
several deficiencies with the assessment programs 
in engineering — basically the lack of direct 
measures or the reliance on indirect measures, 
such as surveys. This highlighted an issue that will 
continue to be more important to engineering 
as ABET becomes less tolerant of programs that 
avoid practicing direct assessment, that fail to 
generate significant faculty participation in doing 
assessment, and that do not use systematic 
processes for improving programming. 
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The feature of PRISM that identified the need 
for direct measures of student learning, (e.g.. 
Component 6 peer-review rubric ofCSU standards) 
has resulted in improvement of the assessment 
program within Engineering over the last year. For 
example all departments in the college recently 
made a significant change in how the senior design 
capstone experience was assessed. Previously, 
surveys where used to assess student learning — 
basically using a self-reporting mechanism. As 
a result of the CSU PRISM process and a 2007 
accreditation visit. Engineering has now adopted 
a collegewide rubric for quantifying the learning 
that is taking place in senior design (see Table 3). 

A working group of five faculty — one from each 
department — acquired the rubric concept from a 
2002 Journal of Engineering Education article and 
substantively modified it to their program needs. 
This rubric is given to the students at the beginning 
of the semester so that it can guide student 
learning. It is also used by the external evaluators 
who review the products that students produce 
in these courses (e.g. design proposals and final 
project reports). The result is a new emphasis on 
direct measures of student learning outcomes. 

As the pressure continues to grow for greater 
student learning accountability in higher education, 
the value of PRISM should become more obvious 
to all colleges and departments. Engineering has 
been resistant to some of these pressures because 
of issues identified above. But the need for change 
has become apparent, and greater implementation 
of PRISM is planned in response to these pressures, 
including having PRISM alignment appear in 
the new Engineering strategic plan. In addition, 
Engineering underwent the CSU program review 
process in fall 2009 (Component 5 integration), 
which included an online self-study with an 
embedded record of its assessment process over 
the last six years. This program review experience 
supported the 2010 hiring of an assessment staff 
person to strengthen departmental utilization 
of PRISM. PRISM provides a mechanism to both 
identify what is required and a platform for planning 
and implementing a comprehensive assessment 
program. 



Conclusion: Benefits, Impact, 
and Challenges of Using 
Improvement Systems 

Benefits of Inter-Institutional Collaboration 

The collaboration ofCSU and UNL provides 
opportunities seldom occurring at large research 
universities. Because they both use a similar online 
assessment and interactive database environment, 
the universities share their best practices in 
program planning and self-evaluation, their student 
learning research instruments (e.g., learning rubrics 
or internship evaluations), and program assessment 
plans, including strategies programs use to develop 
learning for their students. Staff members at 
both institutions envision the time when peer- 
review committee members at one institution can 
review the planning and evaluation effectiveness 
practiced at the other institution. The pathway 
is open for improvement synergies to develop 
as each institution adopts each other's database 
innovations, program impact strategies, and best 
practices for measuring student learning. Sharing 
the system permits each university to discover 
the impact it is having on faculty culture, moving 
it towards acceptance of systematic continuous 
improvement as a positive value. The CSU and 
UNL collaboration recently expanded to include 
additional institutions. 

Evidence of System Impact on Faculty 

Culture 

The following examples from CSU demonstrate 
that PRISM is affecting faculty culture, gradually 
moving it toward a position of embracing, or at least 
using, systematic improvement processes. Some of 
the examples demonstrate the growing planning 
integration that PRISM encourages and a shift 
in departmental reporting away from individual 
faculty vitae towards aggregations of faculty 
research and service data that guide organizational 
goals. Both show that such continuous 
improvement or change management systems can 
help mitigate the negative organizational problems 
of traditional higher education management. 
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Table 3 

CSU Senior Engineering Design Project (meets PRISM peer-review standards for direct 
assessment — Table 1) 





1 


2 


3 


4 


Essentials: 


Elements omitted or show 
little understanding 


All elements used, generally 
understood; design completed 


All elements show depth, 
quality design delivered; 
requirements met 


All elements used skillfully, revised; 
depth of understanding in all parts; 
customer requirements fully met 


Scores: 


Most equal 1 


Most subcategories at 2 


Most scores about 3 


Most scores around 4 


Subcategory: INFORMATION GATHERING— information identified and obtained to support design process and design decisions 


Sources: 


Minimal 


Primarily one 


Multiple, varied, include 
clientele 


Many, widely varied; include state-of-art, 
clientele 


Quality: 


Taken at face value 


Aware of quality variability 


Some quality judgment made 


Information quality judged, 
documented, reviewed 


Frequency: 


Once 


Primarily onetime 


Several distinct times 


Continually acquiring, synthesizing 
information 


Subcategory: PROBLEM DEFINITION— development of design goals and specific requirements that will ensure a successful design 


Client Needs: 


Minimally aware 


Generally aware, considered 


Understood, considered 
broadly 


Internal & external investigated, 
prioritized, refined 


Requirements: 


Few, narrow 


Several, performance related 


Many, varied, technical and 
not 


Encompassing, technical and not, span 
product life 


Definition: 


Vague, unclear 


Clear, specific 


Clear, understandable, 
measurable 


Clear with target values & range, revised 
overtime 


Subcategory: IDEA GENERATION — gathering and creating new ideas and concepts for consideration in development of design 


Climate: 


Critical, stifling 


Supports contribution 


Supports creative thinking 


Stimulating, supportive of creativity 


Methods: 


Single effort 


Primarily single approach 


Multiple methods, sources, 
stimuli 


Methods varied, revised, used multiple 
times 


Types: 


Solution ideas 


Solution ideas 


Solution and design process 
ideas 


Solution and design process ideas 


Subcategory: DESIGN QUALITY— proper use of theory, equations, engineering tools to develop design alternatives 


Quality: 


Incorrect tools, equations, 
theory 


Minor errors in tools, equations 


Technically correct analysis 


Technically correct, creative application 
of tools, equation, theory 


Completeness: 


Incomplete 


Generally complete 


Complete and meets all 
requirements 


Complete, meets requirements, 
exceptionally thorough 


Results: 


Not interpreted 


Interpreted correctly 


Checked, interpreted correctly 


Uncertainty, sensitivity defined; 
reviewed by others 


Subcategory: EVALUATION- 


-using appropriate methods and tools to determine how well concepts meet requirements 


Issues: 


Single issue 


Technical only 


Technical and financial 


Technical, financial, social, environmental 
as appropriate 


Criteria: 


Scarcely identified 


Most requirements considered 


Requirements used, weighted 


All requirements reviewed, refined, 
weighted 


Process/tools: 


Haphazard 


Options loosely ranked 


Options scored and ranked 


Solution optimized by iteration and 
refinement 


Analysis: 


Cursory analysis 


General qualitative analysis 


Proper analytical methods 
and use 


Thorough analysis using best methods 


Subcategory: COMMUNICATION— production of a design report that effectively communicates design process and result to clients 


Clarity: 


Poor organization, clarity 


Organization, clarity lacking in a 
few areas 


Sound organization, 
reasonable clarity 


Exceptional clarity and organization, key 
points emphasized, little unnecessary 
detail, concise summary 


Documentation: 


Incomplete 


Generally complete, some 
details omitted 


Complete 


Complete documentation of design 
process , results, recommendations with 
discussion of alternatives 


Quality: 


Significant errors in 
grammar, style, structure 


A few errors that do not affect 
readability 


Very few errors, appropriate 
style and structure 


Essentially no errors, excellent style and 
structure, key points emphasized, little 
unnecessary detail, concise 



Source: This rubric is based in part on a design found in Davis, D. C., Gentili, K. L.,Trevisan, M. S., & Calkins, D. E. (2002). Engineering 
design assessment process and scoring for program improvement and accountability. Journal of Engineering Education, 91 (2), 21 1 -221 . 
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1 . Even though regular annual reporting initially 
remained weak in four of the eight colleges, 
PRISM participation continues to expand. In 
201 0, the College of Agricultural Sciences 
moved from no PRISM participation in 2008 to 
having three of five department areas actively 
utilizing the planning database. The College 
of Engineering faculty recently completed 
development of a college-wide learning 
outcomes process with rubrics for its design 
capstone experience and hired an assessment 
coordinator to assist implementation utilizing 
PRISM. The School of Education and the 
Department of Forest, Rangeland, and 
Watershed Stewardship redesigned their 
assessment plans in spring 2008. Several of 
the action plan goals emerging from the last 
three cycles of program review are directed 
to organizing increased effort in the PRISM 
assessment process. 

2. University strategic planning metrics include 
PRISM and program review characteristics. 

3. Half of all college strategic plans (four of eight) 
now specify goals related to PRISM use. 

4. New program review guidelines specify use of 
the PRISM annual assessment process. 

5. The University's new program proposal process 
includes PRISM utilization as the means to 
assess student learning outcomes and monitor 
progress on selected program objectives. 

6. Colleges are developing faculty activity 
reporting systems that show research and 
service impact indicators as advocated and 
demonstrated in the PRISM system. 

7. The University Peer Review Committee charge 
has expanded from annual assessment to 
other change management activities, including 
program review and strategic planning. 

8. Departments are using descriptions of PRISM in 
their special accreditation self-studies to meet 
continuous improvement criteria. 

9. NSF grant proposals are beginning to include 
descriptions of the system as the project or 
program assessment component. 

10. PRISM information is now presented annually 
at faculty orientation and department heads 
training. 



11. External program review teams praise PRISM use 
in their evaluation team reports. 

1 2. More and more programs are using learning 
assessment as an instructional strategy to 
deepen learning. For example, capstone courses 
hand out learning rubrics that students use to 
peer review each other's academic work. 

Challenges of Adopting Comprehensive 
Systems 

Institutions can develop their own continuous 
improvement systems or purchase database 
processes from vendors. Flexibility and cost 
management are key features to consider. For 
example, PRISM initially required collaborating 
institutions to operate a ColdFusion server, but now 
UNL and CSU are working together to adopt the 
more commonly used Microsoft .NET framework 
and have the assessment database system become 
an open source product among collaborating 
institutions. The effort is based on the belief that 
these systems should not be controlled, statically 
formatted, or packaged, but should be freely 
evolving processes that can be shared among 
institutions at little or no cost. Institutions should 
be prepared to experience a five-year learning curve 
among a critical mass of faculty, especially at large 
research institutions. In addition, campuses should 
implement sustainability features, such as program 
review integration, regular peer review cycles, uses 
for special accreditation, linkage to department 
head evaluations, engagement of student affairs 
divisions, strategic plan metrics support, and 
development of student expectations for systematic 
program assessment. 

Next Steps in Development 

CSU completed its .NET conversion of PRISM 
in early fall 201 0. This software version will be 
sent to collaborating institutions before 201 1 . 

The assessment and IT staff members from the 
five institutions will work to test this more user- 
friendly and robust software and apply it to their 
unique campus environments. These institutions 
will communicate regularly to make collective 
improvements in the software and inform each 
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other of advances made in their individual campus 
assessment processes, including impact on faculty 
culture. Plans include developing the sharing 
capacity for faculty members at these institutions 
to quickly explore multiple assessment plans 
within their own disciplines to discover what 
other departments use for learning preparation 
strategies and learning research methods and what 
improvements are being identified. 

UNL plans to use its PEARL system to implement 
the assessment of a newly designed general 
education program known as Achievement 
Centered Education. The new .NET software will 
enable faculty to design course-based assessment 
planning that can be organized into subsets of 
general education delivery (e.g., problem solving, 
historical perspectives, or global awareness). 

As CSU approaches its re-accreditation 
site visit in 2014, the University is developing 
PRISM as an evidence file within its self-study 
to show institutional planning and evaluation 
activity, including student learning assessment, 
over a 1 0-year period. As the Higher Learning 
Commission moves to Pathways in 201 5, it will 
require institutions to store evidence files on 
its accreditation Website for periodic Assurance 
Reviews. Another future initiative is to have the 
University faculty activity reporting database 
structure relate to the PRISM program review 
process in 201 1 to reduce duplicative reporting for 
research and service. The next step for developing 
the institution's improvement culture is to work with 
the student affairs division and begin informing 
students of the better examples of student learning 
assessment to strengthen their expectations for 
programmatic quality enhancement. 

Do these University Improvement Systems 
Deliver What the Theorists Advocate? 

Evidence here shows that PRISM and PEARL are 
continuous improvement systems that implement 
many of the quality enhancing characteristics 
recently advocated by Bok, Burke, Massy, and 
others. The systems' regular reporting, historical 
documentation, planning integration, continuous 
peer review, participation monitoring, and reporting 



of process effectiveness combine to reduce the 
impact of the negative factors that retard change 
and innovation in higher education organizations. 

In fact, effective continuous feedback systems can 
encourage program innovation by reducing the risk 
of failure. 

Linking planning and self-evaluation 
information to funding processes remains the 
weakest area of system impact. However, the 
sustained database linking of annual department 
goals and improvements to strategic planning goals 
and metrics should help reinforce this relationship 
over time. At least the relationship becomes more 
visible; and, visibility is a key element in achieving 
accountability. If effectively implemented, these 
active Web-based organizational learning systems 
may become a significant factor in unifying the 
"fragmented university." 
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